Is Coverage a Good Measure of Testing Effectiveness? An Assessment Using Branch Coverage and Random Testing

Most approaches to testing use branch coverage to decide on the quality of a given test suite. The intuition is that covering branches relates directly to uncovering faults. In this article we present an empirical study that applied random testing to 14 Eiffel classes for a total of 2520 hours and recorded the number of uncovered faults and the branch coverage over time. Our results show that: (1) in the tested classes, random testing reaches 93% branch coverage (2) it exercises almost the same set of branches every time, (3) it detects different faults from time to time, (4) during the first 10 minutes of testing while branch coverage increases rapidly, there is a strong correlation between branch coverage and the number of uncovered faults, (5) over 50% of the faults are detected at a time where branch coverage hardly changes and the correlation between branch coverage and the number of uncovered faults is weak. These results provide evidence that branch coverage is not a good stopping criterion for random testing. They also show that branch coverage is not a good indicator for the effectiveness of a test suite.